Skip to content

Use Test.RunnerV2 - #17

Draft
lydell wants to merge 10 commits into
mpizenberg:masterfrom
lydell:runnerv2
Draft

Use Test.RunnerV2#17
lydell wants to merge 10 commits into
mpizenberg:masterfrom
lydell:runnerv2

Conversation

@lydell

@lydell lydell commented Sep 29, 2026 •
edited
Loading

Copy link
Copy Markdown

This PR tracks what it takes to use elm-explorations/test#260 here, with a focus on making this package work with as few changes in elm-test-rs as possible, rather than trying to use Test.RunnerV2 to its fullest. That can be done some other time.

The breaking changes in the Elm API in this PR does not seem to affect elm-test-rs – this package exposes much more than it needs to for some reason.

Companion elm-test-rs PR: mpizenberg/elm-test-rs#148

This is a MAJOR change.

---- ElmTestRunner.Result - MAJOR ----

    Removed:
        fromExpectations : List String -> List Expectation -> TestResult


---- ElmTestRunner.SeededRunners - MAJOR ----

    Added:
        type SeededRunners
        getKind :
            ElmTestRunner.SeededRunners.SeededRunners
            -> ElmTestRunner.SeededRunners.Kind
        getTestsCount : ElmTestRunner.SeededRunners.SeededRunners -> Basics.Int

    Removed:
        type alias SeededRunners =
            Result String { kind : Kind, runners : Array Runner }

    Changed:
      - run : Int -> Array Runner -> Maybe TestResult
      + run :
            Basics.Int
            -> ElmTestRunner.SeededRunners.SeededRunners
            -> Maybe.Maybe
                   (Task.Task Basics.Never ElmTestRunner.Result.TestResult)

@lydell
Breaking changes not related to RunnerV2
2e28747
@lydell
Use Test.RunnerV2 instead of Test.Runner
62686f0 
With a focus on making as few breaking changes as possible, rather than
trying to use `Test.RunnerV2` to its fullest. That can be done some
other time.
@lydell lydell mentioned this pull request Sep 29, 2026
Draft
lydell
lydell commented Sep 29, 2026
View reviewed changes
Comment thread src/ElmTestRunner/SeededRunners.elm Resolved
mpizenberg
mpizenberg reviewed Oct 6, 2026
View reviewed changes

@mpizenberg mpizenberg left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems there will be a problem with error reporting in case a failure happen in unit tests. The ConsoleColor code calls "List.map2 (failureToText usColor) failures distributionReports". But with the changes here, the "distributionReports" value would be empty, and the zipping of map2 will thus be empty, and thus ignoring all failures. We should probably pass "NoDistribution ()" for unit tests, or stop zipping both with map2.

There might be exercism-specific checks to do. The order of the tests results is be important I think if I remember correctly, so changes related to the split of unit and fuzz might have an impact. To be checked.

I see there are changes related to timing and debug logging in the runnerv2. This is cool, and eventually we might want to change it here, to retrieve it instead of the workaround we currently do in elm-test-rs supervisor.

Claude detected there might be an issue with how the runs count is handled in "fuzzWith", and probably being ignored in V2. I didn’t check. It also thinks there might an issue with console logging using unbuffered mode, because it calles "console.error(...)" directly, and it might change how elm-test-rs detects logs. I haven’t checked, but in any case, it’s probably better to change elm-test-rs and elm-test-runner to work with the new way to collect debug logs (and timing).

Comment thread src/ElmTestRunner/SeededRunners.elm Resolved
Comment thread src/ElmTestRunner/SeededRunners.elm Resolved
Comment thread src/ElmTestRunner/SeededRunners.elm Resolved
@lydell lydell mentioned this pull request Oct 6, 2026
Draft
@lydell

lydell commented Oct 6, 2026

Copy link
Copy Markdown
Author

We should probably pass "NoDistribution ()" for unit tests, or stop zipping both with map2.

I now pass NoDistribution () for unit tests in 55c62f4

There might be exercism-specific checks to do. The order of the tests results is be important I think if I remember correctly, so changes related to the split of unit and fuzz might have an impact.

Did the tests have a defined order previously? 🤔 Let me know once you know more about Exercism!

Claude detected there might be an issue with how the runs count is handled in "fuzzWith", and probably being ignored in V2.

Thanks, this was a bug in the RunnerV2 branch. Fixed in elm-explorations/test@8292c19

It also thinks there might an issue with console logging using unbuffered mode, because it calles "console.error(...)" directly, and it might change how elm-test-rs detects logs.

Good catch. The best solution is to make a few changes for this in elm-test-rs. I’ve made a PR there as well: mpizenberg/elm-test-rs#148

@mpizenberg
Remove duration and logs from incoming results messages
f4b4d4d 
Signed-off-by: Matthieu Pizenberg <matthieu.pizenberg@gmail.com>
@mpizenberg

mpizenberg commented Oct 8, 2026

Copy link
Copy Markdown
Owner

Made a couple changes here and in elm-test-rs. Looks good to me.

Test order isn't a problem, because we re-sort results by test name into source-file order, doing some parsing. The test-code extractor expects the old Test.fuzz argument order (description in third position) and so will break once the track moves to the new elm-explorations/test 3.x API. It can be fixed afterwards because we pin the version of elm-test-rs in exercism containers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mpizenberg mpizenberg mpizenberg left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lydell @mpizenberg